Ichnaea: Statistical Parallel Profiling for Clusters

In this paper, the current state of hardware profiling on networked clusters is examined in some depth, with some conjecture about the near future. Next, an implementation of a simple Unix command-line tool Ichnaea is given. Ichnaea automates the process of clusterwide profiling. Finally, we examine opportunities for extension and further work. Profiling Background Traditionally, a profiler is a piece of code that runs alongside a target program while logging the interesting actions (for some user-defined concept of interesting action) of the target program. Many profilers also include a tool that generates a human-readable report from these log files. There is an important distinction to be made between deterministic profilers and statistical profilers: a deterministic profiler records each action taken by a program (variables initialized, memory allocations, function calls, exceptions, etc.) with full precision and consistency, whereas a statistical profiler uses a sampling-based method to log snapshots of the stack frame and instruction pointer during program execution. Profilers have historically been used in a number of ways, but the two most common use cases are debugging and performance optimization. Debugging profilers are (by necessity) deterministic, but there exist both deterministic and statistical profilers for optimization. It is usually the case that deterministic profilers incur greater overhead costs than statistical